1 About

Code copyright (c) 2024 the author

License:

To cite:

Code:

3 Data

This analysis uses the New Zealand Electricity Authority’s Generation data.

Values are in kWh per half hour https://www.emi.ea.govt.nz/Wholesale/Datasets/Generation/Generation_MD

As far as we can work out this data does not include distributed (i.e. non-grid connected) generation such as small scale wind, solar, hydro, biomass etc which is connected to the LV network. This means the EA data is likely to underestimate total generation and potentially underestimate the proportion of total generation that is renewable. It is possible that this could be fixed using embedded wind & solar generation data from the metered embedded generation data although this may still not include household level (micro)generation.

rmdParams$olderThan <- 7

If the data we have previously downloaded is more than 7` days old, re-download.

rmdParams$dataPath <- "~/Dropbox/data/NZ_ElecAuth/processed/"

# load all the files we have using rbindlist
# probably ought to do some filtering here
filesToLoad <- list.files(rmdParams$dataPath, pattern = ".csv", 
                          full.names = TRUE)

orig_DT <- rbindlist(lapply(filesToLoad, fread))
setnames(orig_DT, "rDateTime", "dv_dateTime")

message("Original data range from: ", min(orig_DT$dv_dateTime, na.rm = TRUE))
## Original data range from: 2020-01-01 00:15:00
message("...to: ", max(orig_DT$dv_dateTime , na.rm = TRUE))
## ...to: 2024-07-31 23:45:00
# make wide for ease of aggregation ----
# sum over fuel code
nzGenMix_wide_dt <- dcast(orig_DT[!is.na(dv_dateTime)], dv_dateTime ~ Fuel_Code, value.var = "kWh", fun.aggregate = sum)

# add derived variables used later ----
addDerivedVars <- function(dt){
  dt[, dv_year := lubridate::year(dv_dateTime)]
dt[, dv_date := lubridate::as_date(dv_dateTime)]
dt[, dv_month := lubridate::month(dv_dateTime)]
dt[, dv_hour := lubridate::hour(dv_dateTime)]
dt[, dv_hms := hms::as_hms(dv_dateTime)]
dt <- gridCarbon::add_season(dt, 
                                 dateVar = "dv_dateTime",
                                 h = "S")
dt <- gridCarbon::add_peakPeriod(dt, 
                                      dateTime = "dv_dateTime")
  return(dt)
}

nzGenMix_wide_dt <- addDerivedVars(nzGenMix_wide_dt)
nzGenMix_dt <- addDerivedVars(orig_DT[!is.na(dv_dateTime)])

#message("Remove incomplete years to avoid weird things in plots.")

message("Filtered data range from: ", min(nzGenMix_wide_dt$dv_dateTime))
## Filtered data range from: 2020-01-01 00:15:00
message("...to: ", max(nzGenMix_wide_dt$dv_dateTime))
## ...to: 2024-07-31 23:45:00
rmdParams$plotCap <- paste0("Data: NZ EA Generation ",
                   min(nzGenMix_wide_dt$dv_date), " - ",
                   max(nzGenMix_wide_dt$dv_date),
                            "\nPlot: @dataknut \nCode: https://github.com/dataknut/gridCarbon")

# > recent date cut ----
recentDateCut <- max(nzGenMix_wide_dt$dv_date - (6*30)) # roughly 6 months

Note:

  • the data may cover more years than we need
  • the data may contain partial years - BEWARE incomplete years in plots using annual totals or means across all months.

4 Recreating DrSimEvans’ plot

Inspired by https://twitter.com/DrSimEvans/status/1508409309775994892

Which explored the relationship between % renewable and fossil-fuel generation for Great Britain (see also https://dataknut.github.io/gridCarbon/gbGenMixTrends.html)

This looks like daily data.

  • Solar + wind (% of gen) vs coal + gas (and % of gen))
# add together the % and totals we want (half-hourly)

nzGenMix_wide_dt[, dv_total := Coal + Diesel + Gas + Geo + Hydro + Solar + Wind + Wood]
nzGenMix_wide_dt[, dv_coal_gas := Coal + Gas]
nzGenMix_wide_dt[, dv_coal_gas_pc := 100 * dv_coal_gas/dv_total]
nzGenMix_wide_dt[, dv_solar_wind := Solar + Wind]
nzGenMix_wide_dt[, dv_solar_wind_pc := 100 * dv_solar_wind/dv_total]

# keep the vars we want for clarity
temp <- nzGenMix_wide_dt[, .(dv_dateTime, dv_coal_gas, dv_solar_wind,
                    dv_coal_gas_pc, dv_solar_wind_pc, dv_total)]

temp[, dv_date := lubridate::date(dv_dateTime)]

# aggregate to daily data for plotting
plotDT <- temp[,
               .(mean_dv_solar_wind_pc = mean(dv_solar_wind_pc),
                 mean_dv_coal_gas_pc = mean(dv_coal_gas_pc),
                 total_dv_coal_gas = sum(dv_coal_gas),
                 total_dv_solar_wind = sum(dv_solar_wind),
                 total_dv_total = sum(dv_total),
                 nObs = .N), # to check for days with < 48 half hours
               keyby = .(dv_date)
               ]
plotDT[, dv_year := lubridate::year(dv_date)] # for plots
plotDT[, total_dv_coal_gas_pc := 100 * total_dv_coal_gas/total_dv_total] # daily %
plotDT[, total_dv_solar_wind_pc := 100 * total_dv_solar_wind/total_dv_total]

message("Check for days with less than 48 hours - this will be truncated data due to DST breaks. We hate DST breaks")
## Check for days with less than 48 hours - this will be truncated data due to DST breaks. We hate DST breaks
table(plotDT$nObs)
## 
##   46   48 
##    4 1670

Figure 4.1 shows the mean half-hourly % generation by each type per day. This is slightly convoluted - it is the mean of the sum of the 48 daily half-hourly values. Unfold the code above for clarity.

The smoothed curves are estimated for each year. The lines terminate at the maximum value for the year. I’m still trying to decide if they tell us anything useful.

ggplot2::ggplot(plotDT[dv_year > 2011], aes(x = mean_dv_solar_wind_pc, 
                            y = mean_dv_coal_gas_pc,
                            colour = as.factor(dv_year),
                            alpha = dv_year)) +
  geom_point() +
  geom_smooth() +
  scale_colour_viridis_d(name = "Year") +
  guides(alpha = "none") +
  labs(x = "Solar & wind (mean % of half-hourly generation per day)",
       y = "Coal & gas (mean % of half-hourly generation per day)",
       caption = rmdParams$plotCap)
## `geom_smooth()` using method = 'loess' and formula = 'y ~ x'
Mean half-hourly % generation by each type per day

Figure 4.1: Mean half-hourly % generation by each type per day

# save it
ggplot2::ggsave(filename = "meanNZrenewablesVsfossilHalfHourPC.png", 
                path = rmdParams$plotPath,
                height = 5)
## Saving 7 x 5 in image
## `geom_smooth()` using method = 'loess' and formula = 'y ~ x'

Figure 4.2 shows the percentage of daily generation by type. This is less convoluted as it is the sum of generation per day for the two categories (solar + wind vs gas + coal) as a % of total daily generation.

Again the smoothed curve is estimated for each year.

ggplot2::ggplot(plotDT[dv_year > 2011], aes(x = total_dv_solar_wind_pc, 
                            y = total_dv_coal_gas_pc,
                            colour = as.factor(dv_year),
                            alpha = dv_year)) +
  geom_point() +
  geom_smooth() +
  scale_colour_viridis_d(name = "Year") +
  guides(alpha = "none") +
  labs(x = "Solar & wind (% of total daily generation)",
       y = "Coal & gas (% of total daily generation)",
       caption = rmdParams$plotCap)
## `geom_smooth()` using method = 'loess' and formula = 'y ~ x'
Percentage of daily generation by type

Figure 4.2: Percentage of daily generation by type

ggplot2::ggsave(filename = "dailyNZpcGenMix.png", 
                path = rmdParams$plotPath,
                height = 5)
## Saving 7 x 5 in image
## `geom_smooth()` using method = 'loess' and formula = 'y ~ x'

4.1 Half-hourly versions of the plot

Just cos we can… helpfully split into ‘peak’ and ‘off peak’ periods.

Peak period definitions:

  • Morning 07:00 - 09:00
  • Daytime 09:00 - 16:00
  • Evening 16:00 - 21:00
  • Night - all other times

Again the smoothed curve is estimated for each year (and demand period).

ggplot2::ggplot(nzGenMix_wide_dt[dv_year > 2011], aes(x = dv_solar_wind_pc, 
                            y = dv_coal_gas_pc,
                            alpha = dv_year,
                            colour = as.factor(dv_year))) +
  geom_point() +
  facet_wrap(. ~ dv_peakPeriod) +
  geom_smooth() +
  scale_colour_viridis_d(name = "Year") +
  guides(alpha = "none") +
  labs(x = "Solar & wind (% of half-hourly generation)",
       y = "Coal & gas (% of half-hourly generation)",
       caption = rmdParams$plotCap)
## `geom_smooth()` using method = 'gam' and formula = 'y ~ s(x, bs = "cs")'
Percentage of half-hourly generation by type

Figure 4.3: Percentage of half-hourly generation by type

ggplot2::ggsave(filename = "halfHourlyNZ_PCgenByPeakPeriod.png", 
                path = rmdParams$plotPath,
                height = 5)
## Saving 7 x 5 in image
## `geom_smooth()` using method = 'gam' and formula = 'y ~ s(x, bs = "cs")'

7 Fuel types and time of day ‘demand’

Do we see a relationship between generation fuels and peak demand? This will be mediated by the way the electricity market works.

We may find wind curtailment (not visible here) at low demand periods where geothermal can’t be shut off.

First, what is the general average shape of overall generation?

‘Peak periods’ shown as shaded rectangles

# orginal values are kWh
plotDT <- nzGenMix_wide_dt[, .(mean_total_GWh = mean(dv_total/1000000),
                               mean_renewables_GWh = mean(RENEWABLE/1000000),
                               mean_renewables_pc = mean(RENEWABLE_perc),
                               mean_hydro_GWh = mean(Hydro/1000000),
                               mean_hydro_pc = mean(HYDRO_perc),
                               mean_coal_GWh = mean(Coal/1000000),
                               mean_coal_pc = mean(COAL_perc),
                               mean_gas_GWh = mean(Gas/1000000),
                               mean_gas_pc = mean(GAS_perc)), keyby = .(dv_hms, dv_year, dv_season, dv_peakPeriod)]

make_profilePlot <- function(dt, yVar = "mean_renewables_GWh", yVarLab = "change me"){
  plotDT <- dt[, .(x = dv_hms, y = get(yVar), alpha = dv_year, 
                   group = dv_year,
                   facet1 = dv_season,
                   colour = dv_peakPeriod)]
  p <- ggplot2::ggplot(plotDT, aes(x = x, 
                                   y = y,
                            alpha = alpha,
                            group = group)) +
    geom_rect(aes(xmin=hms::as_hms("07:00:00"), xmax=hms::as_hms("09:00:00"),
                  ymin=min(plotDT$y), ymax=max(plotDT$y)),
                              fill='pink', alpha=0.1) +
        geom_rect(aes(xmin=hms::as_hms("16:00:00"), xmax=hms::as_hms("21:00:00"),
                  ymin=min(plotDT$y), ymax=max(plotDT$y)),
                              fill='pink', alpha=0.1) +
  geom_line() +
  scale_alpha_continuous(name = "Year") +
  facet_wrap(facet1 ~ .) +
    labs(x = "Time of day",
       y = yVarLab,
       caption = rmdParams$plotCap) +
  theme(axis.text.x = element_text(angle = 45, hjust=1)) +
    scale_x_time(breaks = scales::breaks_width("120 min")) 
  return(p)
}

make_profilePlot(plotDT, "mean_total_GWh", "Mean total (GWh)")
Mean half-hourly generation by year and season

Figure 7.1: Mean half-hourly generation by year and season

Contribution of renewables…

make_profilePlot(plotDT, "mean_renewables_GWh", "Mean renewables (GWh)")
Mean half-hourly % renewable generation by year and season

Figure 7.2: Mean half-hourly % renewable generation by year and season

make_profilePlot(plotDT, "mean_renewables_pc", "Mean % renewables")
Mean half-hourly % renewable generation by year and season

Figure 7.3: Mean half-hourly % renewable generation by year and season

What is hydro’s contribution?

make_profilePlot(plotDT, "mean_hydro_GWh", "Mean hydro (GWh)")
Mean half-hourly % hydro generation by year and season

Figure 7.4: Mean half-hourly % hydro generation by year and season

make_profilePlot(plotDT, "mean_hydro_pc", "Mean % hydro")
Mean half-hourly % hydro generation by year and season

Figure 7.5: Mean half-hourly % hydro generation by year and season

What is coal’s contribution?

make_profilePlot(plotDT, "mean_coal_GWh", "Mean coal (GWh)")
Mean half-hourly % coal generation by year and season

Figure 7.6: Mean half-hourly % coal generation by year and season

make_profilePlot(plotDT, "mean_coal_pc", "Mean % coal")
Mean half-hourly % coal generation by year and season

Figure 7.7: Mean half-hourly % coal generation by year and season

What is gas’s contribution?

make_profilePlot(plotDT, "mean_gas_GWh", "Mean gas (GWh")
Mean half-hourly % gas generation by year and season

Figure 7.8: Mean half-hourly % gas generation by year and season

make_profilePlot(plotDT, "mean_gas_pc", "Mean % gas")
Mean half-hourly % gas generation by year and season

Figure 7.9: Mean half-hourly % gas generation by year and season

Relationship between hydro and overall generation

ggplot2::ggplot(nzGenMix_wide_dt, aes(y = Hydro/1000000, x = dv_total/1000000, colour = dv_peakPeriod)) +
  geom_point() +
  facet_grid(dv_year ~ dv_season) +
  labs(y = "Hydro GWh per half hour",
       x = "Total GWh per half hour")

YMMV on 7.10

plotDT <- nzGenMix_wide_dt[, .(mean_renewables_GWh = mean(RENEWABLE/1000000),
                 mean_generation_GWh = mean(dv_total/1000000)),
             keyby = .(dv_year, dv_hms, dv_peakPeriod, dv_season)]

ggplot2::ggplot(plotDT, aes(x = mean_generation_GWh , y = mean_renewables_GWh,
                                         colour = dv_peakPeriod)) +
  geom_point() +
  scale_color_discrete(name = "Period") +
  facet_grid(dv_season ~ dv_year) +
  labs(x = "Mean half-hourly generation (GWh)",
       y = "Mean half-hourly renewables (GWh)",
       caption = rmdParams$plotCap)
Mean renewables vs Mean total generation

Figure 7.10: Mean renewables vs Mean total generation

8 Summary

That’s it.

You might want to look at recent academic research on this topic:

  • (Staffell 2017)
  • (Staffell and Pfenninger 2018)

9 Annex

9.2 Data descriptors

9.2.1 NZ gen mix data

skimr::skim(nzGenMix_wide_dt)
Table 9.1: Data summary
Name nzGenMix_wide_dt
Number of rows 80344
Number of columns 32
Key dv_dateTime
_______________________
Column type frequency:
Date 1
difftime 2
factor 2
numeric 26
POSIXct 1
________________________
Group variables None

Variable type: Date

skim_variable n_missing complete_rate min max median n_unique
dv_date 0 1 2020-01-01 2024-07-31 2022-04-16 1674

Variable type: difftime

skim_variable n_missing complete_rate min max median n_unique
dv_hms 0 1 900 secs 85500 secs 42300 secs 48
hms 0 1 900 secs 85500 secs 42300 secs 48

Variable type: factor

skim_variable n_missing complete_rate ordered n_unique top_counts
dv_season 0 1 FALSE 4 Aut: 22080, Win: 20592, Sum: 20208, Spr: 17464
dv_peakPeriod 0 1 FALSE 5 Ear: 23436, Day: 23436, Eve: 13392, Lat: 13384

Variable type: numeric

skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100 hist
Coal 0 1 102551.40 95511.68 0.00 0.00 85433.00 174755.50 365201.00 ▇▅▃▂▁
Diesel 0 1 350.50 3382.94 0.00 0.00 0.00 0.00 79213.00 ▇▁▁▁▁
Gas 0 1 257958.57 124946.18 18133.00 171104.75 250195.00 339700.00 646409.00 ▃▇▆▂▁
Geo 0 1 427833.33 35953.24 224851.00 410573.50 431223.50 447368.25 550997.00 ▁▁▅▇▁
Hydro 0 1 1374752.44 346190.15 465067.00 1105091.00 1393511.50 1623827.75 2363701.00 ▂▆▇▅▁
Solar 0 1 193.34 1345.51 0.00 0.00 0.00 0.00 19086.00 ▇▁▁▁▁
Wind 0 1 153262.07 98054.21 0.00 70677.00 139026.00 223276.75 477396.00 ▇▇▅▂▁
Wood 0 1 13027.73 5460.84 0.00 11880.00 15231.00 16670.00 19561.00 ▂▂▁▆▇
dv_year 0 1 2021.82 1.34 2020.00 2021.00 2022.00 2023.00 2024.00 ▇▇▇▇▅
dv_month 0 1 6.20 3.40 1.00 3.00 6.00 9.00 12.00 ▇▆▆▅▆
dv_hour 0 1 11.50 6.92 0.00 5.00 11.00 17.00 23.00 ▇▇▆▇▇
dv_total 0 1 2329929.38 404976.50 1324732.00 1990030.50 2361135.00 2608821.75 3626574.00 ▂▆▇▂▁
dv_coal_gas 0 1 360509.97 183929.44 19460.00 212909.75 343146.50 495899.25 962020.00 ▆▇▇▂▁
dv_coal_gas_pc 0 1 15.18 6.77 1.02 9.77 15.23 20.39 35.68 ▃▇▇▅▁
dv_solar_wind 0 1 153455.41 98092.74 0.00 70841.75 139161.50 223397.00 477396.00 ▇▇▅▂▁
dv_solar_wind_pc 0 1 6.77 4.50 0.00 3.08 5.98 9.79 24.57 ▇▆▃▁▁
RENEWABLE 0 1 1969068.91 337707.31 1143537.00 1700021.50 1989244.00 2216382.00 3062302.00 ▂▆▇▃▁
RENEWABLE_perc 0 1 84.81 6.79 64.32 79.60 84.77 90.23 98.98 ▁▅▇▇▃
WIND_perc 0 1 6.77 4.50 0.00 3.07 5.97 9.79 24.57 ▇▆▃▁▁
SOLAR_perc 0 1 0.01 0.05 0.00 0.00 0.00 0.00 0.74 ▇▁▁▁▁
HYDRO_perc 0 1 58.50 8.22 25.83 53.36 58.74 63.94 83.17 ▁▂▇▆▁
GEO_perc 0 1 18.95 3.84 9.56 16.06 18.14 21.67 34.20 ▂▇▅▂▁
GAS_perc 0 1 10.91 4.63 0.88 8.00 11.01 14.10 25.10 ▃▇▇▃▁
COAL_perc 0 1 4.27 3.93 0.00 0.00 3.64 7.03 20.65 ▇▃▂▁▁
ratio 0 1 0.12 0.10 0.00 0.05 0.10 0.17 0.78 ▇▃▁▁▁
CARBON_INTENSITY 0 1 187.18 42.25 100.15 151.21 184.01 218.09 339.08 ▅▇▆▂▁

Variable type: POSIXct

skim_variable n_missing complete_rate min max median n_unique
dv_dateTime 0 1 2020-01-01 00:15:00 2024-07-31 23:45:00 2022-04-17 80344

9.3 R environment

Packages etc:

  • base R (R Core Team 2016)
  • bookdown (Xie 2016a)
  • data.table (Dowle et al. 2015)
  • ggplot2 (Wickham 2009)
  • here (Müller 2017)
  • hms (Müller 2018)
  • knitr (Xie 2016b)
  • lubridate (Grolemund and Wickham 2011)
  • rmarkdown (Allaire et al. 2018)

References

Allaire, JJ, Yihui Xie, Jonathan McPherson, Javier Luraschi, Kevin Ushey, Aron Atkins, Hadley Wickham, Joe Cheng, and Winston Chang. 2018. Rmarkdown: Dynamic Documents for r. https://CRAN.R-project.org/package=rmarkdown.
Dowle, M, A Srinivasan, T Short, S Lianoglou with contributions from R Saporta, and E Antonyan. 2015. Data.table: Extension of Data.frame. https://CRAN.R-project.org/package=data.table.
Grolemund, Garrett, and Hadley Wickham. 2011. “Dates and Times Made Easy with lubridate.” Journal of Statistical Software 40 (3): 1–25. http://www.jstatsoft.org/v40/i03/.
Müller, Kirill. 2017. Here: A Simpler Way to Find Your Files. https://CRAN.R-project.org/package=here.
———. 2018. Hms: Pretty Time of Day. https://CRAN.R-project.org/package=hms.
R Core Team. 2016. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Staffell, Iain. 2017. “Measuring the Progress and Impacts of Decarbonising British Electricity.” Energy Policy 102 (March): 463–75. https://doi.org/10.1016/j.enpol.2016.12.037.
Staffell, Iain, and Stefan Pfenninger. 2018. “The Increasing Impact of Weather on Electricity Supply and Demand.” Energy 145 (February): 65–78. https://doi.org/10.1016/j.energy.2017.12.051.
Wickham, Hadley. 2009. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. http://ggplot2.org.
Xie, Yihui. 2016a. Bookdown: Authoring Books and Technical Documents with R Markdown. Boca Raton, Florida: Chapman; Hall/CRC. https://github.com/rstudio/bookdown.
———. 2016b. Knitr: A General-Purpose Package for Dynamic Report Generation in r. https://CRAN.R-project.org/package=knitr.